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PATENT APPLICATION 
Attorney Docket No: 1 9904-006 (BREESE-6) CON 

OPTIMAL VIDEO DECODER BASED ON MPEG-TYPE STANDARDS 

CLAIM OF PRIORITY 
This application claims priority under 35 U.S.C. § 1 19(a) to French patent 
application 99 07443, filed June 11, 1999. 

TECHNICAL FIELD OF THE INVENTION 
The invention concerns the display of animated images, in particular 
decompression of digital data which incorporate these images using optimized methods. 

BACKGROUND OF THE INVENTION 
With the appearance of the most recent digital technologies, and the ever- 
increasing need for speed and storage space, compression cannot be avoided for 
mainstream applications. Examples include digital cameras which code images in JPEG, 
digital camcorders which compress DV format sequences, an M-JPEG derivative, or 
digital television and DVD, which have adopted the MPEG-2 compression format, in 
addition, of course, to the Internet, in which images and sequences are sent in compressed 
form. 

In certain cases, the user requires very high quality (photo, camcorder), implying 
very low rates of compression. In other cases, excessive transfer time prevents acceptable 
quality. It is, therefore, necessary to improve sequence decoding to permit either better 
quality at an equivalent rate, or a weaker rate with equal or superior quality. 

Different animated image compression standards have been proposed, but only the 
MPEG standard has really taken hold. This standard for the compression and 
decompression of animated images leads to the appearance of block effects. 



For example, European patent EP539833 concerns a process designed to produce 
a compressed video data representation which can be displayed on a video screen after 
decompression according to a number of hierarchical scales of image and/or quality 
resolution, including phases which consist of: 

providing video image element data signals indicating block units in space or 
macro-blocks, which associate the information concerning the compressed image 
data with a group of coding attributes, including coding decisions, movement 
compensation vectors, and quantification parameters, and 
producing for each of these macro-blocks a macro block which is placed on the 
corresponding scale for each scale of this multiplicity so that the same coding 
attributes are shared by these scaled macro-blocks. 

Methods enabling decompression errors to be corrected have been proposed by 
previous researchers. These methods primarily concern techniques used after the process 
of image decompression itself and slow down this compression. These methods do not 
take the quantifier into account and do not permit the binary frame to be retained after 
recompression, which has the effect of degrading image quality with each compression. A 
goal of the invention is to propose a process for improving image quality during 
decompression. 

SUMMARY OF THE INVENTION 
The invention concerns a process for decompression of compressed animated 
images with a method including treatment of images in blocks and containing a digital 
data recomposition phase defining predefined forms, a phase modeling the movement of 
these forms using a process of prediction, interpolation and temporal compensation, an 
image composition phase from reconstructed elements of JPEG or MPEG type motion. 
The form recomposition phase includes a process for separating fixed forms from mobile 
forms, a process for recording digital data corresponding to the fixed forms treated by a 
filter which is not separable from the processes implemented in the recomposition phase 



in a first specific memory unit and digital data corresponding to mobile forms in a second 
specific memory unit. 

Beneficially, the digital filter is irreducible and does not contain dissociable 
filters. In one variant, the filter eliminates the block effect on the background image. It 
can regularize the background image. Beneficially, the quantification interval used during 
compression of the background image is stored and projected on the quantification 
interval. 

In one variant, reconstruction of the elements uses quantification parameters 
previously defined by the coder during image compression. These parameters are linked 
to the image photography methods and permit decompression to be adapted based on 
these methods. This permits taking account of the compression characteristics and 
improving image decompression. In one variant, the quantification parameters are defined 
by the transfer function of the methods of acquisition and storage of animated images. 

Beneficially, a second digital filter separates and identifies the mobile elements 
into mobile objects moving in a sequence, in accordance with the evolution of 
predetermined digital criteria, such as the geometry of mobile objects, movement of 
mobile objects, or spatial segmentation of mobile objects. Temporal averaging can also 
take place with compensation for the movement of each identified object. In one variant, 
the filter eliminates the block effect from objects. According to this process, the objects 
identified can also be regularized. 

Beneficially, the quantification interval serving to compress the animated 
sequence is stored and is projected on the quantification interval. During display, the 
mobile objects and averaged representation are superimposed in the fixed image time. 

Preferably, the parameters specific to each object identified are stored separately 
in order to treat each object differently. 

The invention also concerns a device for decompression of compressed animated 
images with a method including image treatment in blocks and a digital data 
recomposition phase defining predefined forms, a movement modeling stage of these 



forms using methods for prediction, interpolation and temporal compensation, an image 
composition phase from reconstructed elements of JPEG or MPEG type motion. It 
includes methods for separating fixed forms from mobile forms, and methods for 
recording digital data corresponding to the fixed forms treated by a filter which is not 
separable from the methods implemented in the recomposition phase in a first specific 
memory unit, and digital data corresponding to mobile forms in a second specific memory 
unit. 

This device also beneficially includes irreducible digital filtering methods, which 
cannot be decomposed into a sequence of filters independent of one another. 

It includes, preferably, storage methods for the types of images compressed. 

One variant of this device includes a detachable medium and can be made with an 
independent chip or a graphics memory card that can be inserted into a computer. This 
device can also be inserted without being separated into a computer or into any type of 
electronic apparatus permitting image display. 

This device can also be comprised using an independent software module from 
the software present in a calculator memory. 

The invention consists of a decoding process for reducing both the block effects 
and defects linked to degradation of the sequence media. 

This method treats the problem spatially and temporally, obtaining significant 
improvement in sequence quality, and is based on two ideas: 

simultaneously treating the problems of block suppression and movement 

segmentation, 

integrating the notion of object, in order to permit a different approach for 
treatment of the background and of each object. 

In addition, this method presents the particular characteristic of effectively 
treating the problem of "drop-out", independent of the original sequence format: the 
image blocks lost to acquisition by the camera, or during transmission are perfectly 
restored; and defects such as abrasions and threads are removed from digital film. In 



addition, it is possible to integrate the process of accounting for the objective transfer 
function of the camera, or the projector, into the decoding process to obtain a more 
precise restitution. Finally, the scheme proposed remains valid within the framework of 
MPEG-4 decoding. 

The method proposed uses an object approach with two distinct phases. Firstly, 
the sequence background is isolated and the block effects in it are suppressed. The objects 
are slowly isolated from the background, benefitting from a more precise representation 
at each stage. Each object is then treated independently, according to its own 
characteristics, and then finally projected on the estimated background image to 
reconstruct the sequence. 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a flow chart of the method of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 
FIG. 1 represents the stages of this process. 

A first stage (1) consists of estimation and background image treatment, in 
addition to identification of the mobile elements. A map of these mobile elements is 
transmitted by the treatment of these elements. 

Stage (2) consists of pretreating this map by labeling and completing each 
element. 

Stage (3) consists of spatial segmentation of the different elements permitting 
identification of the different mobile objects. This stage also permits estimation of this 
movement, and follow-up of objects during the sequence. 

Stage (4) specifically treats each object identified according to the methods 
explained below. 

Stage (5) consists of the reconstruction of the sequence which permits obtaining 
the decoded sequence. 



Estimation of the background is considered to be an inverse problem. Let pk be N 
images of the MPEG or M-JPEG sequence containing the block effects. The background 
is simultaneously estimated, and the sequence of mobile objects, termed c*. 

We want c*=0 if the point belongs to a mobile object, otherwise c*=l . We look for 

/* = argmin(A) 

with as criterion 

J\if, Ci,...,c N ) = J\(f 9 Ci,..., c N ) +X 2 J 2 (j) + y 2 Ji(ck) 
with 



k=\ *=i 



10 which causes spatiotemporal segmentation using N consecutive images of the 

sequence, and 

m= J M|v/D) (i) 

Mc k )= J *2(||Vc*||)) (2) 

the regularization terms which a priori contain the solution. §\ and are potential 
15 functions which maintain the discontinuities in the image. Parameter a c determines the 

importance granted to the background: the smaller a c is, the more mobile objects are 
detected. 

Relative to Jj(f), if pk is far away from the current estimate /, c* must be small: 
the object is moving. 

20 This comprises a traditional approach for spatiotemporal segmentation of 

sequences. However, this method does not affect the block effects resulting from the 
DCT, and does not take into consideration coder characteristics. The treatment specific to 
the invention solves this problem. 

To take account of the quantifier and simultaneously suppress the block effects 

25 during extraction of / background, the new criterion is minimized: 
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J(f,c h ... f c N ) = j x (f t c,,... f c N ) +x x 2 j 2 (J) + yi V 3 (/) + thV 3 (/) + m 2 J 3 (/) (3) 



where if is the transformation into wavelets, *F a potential function, and 8 a 
threshold dependent upon block effect amplitude. The value of c/ specifies which wavelet 
coefficients are to be thresholded. A soft thresholding in the spatio-frequential wavelet 
area is then performed. 

Specific knowledge of the quantification matrix used during coding permits each 
pixel from the reconstructed sequence to be restricted to an interval corresponding to the 
quantification interval. 

Quantification is a discretization operation which transforms a continuous group 
of sample values to a discrete group. It can be performed on a single sample at the same 
time (scalar quantification) or several samples assembled in blocks (vector 
quantification). The restriction corresponding to this projection is: 



with D being the DCT operator,/?* the quantified DCT coefficient, q the 
quantification step for the pixel considered. 



with 




(4) 





dJx\ 

The /* optimal solution for this minimization problem is found for — r = 0, 

0 



equivalent to: 



X c k 2 (f-p k )-X x 2 div 



with 



r 



k(/) = 



I |v/| 



v/ 



l 0 



+ Tli 2 R r 



fl 



if Df<p k -± 

if zy>p*+| 

if Dfe[p k -l;p k +l] 



( 



Optimal objective cards are obtained for — — = 0, yielding the following 



equation: 



N N 

2 Ckif-Pkf+Q*^ (c k - 1)- Yi 2 ^v 



*=1 



<K(M) 



Va = 0 



(6) 



I |Va| 

The problem is solved with two successive optimizations: 
Minimization of (5) in /, for c* given, /* 
Minimization of (6) in c*, for /* given -> c** 

These two optimizations are then iterated by searching for a new /* background, 
followed by new c*, until the convergence of the solution. 

The criterion is resolved using a semi-quadratic resolution algorithm described in 
the article Deterministic Edge-Preserving Regularization in Computed Imaging, 5(12) 
IEEE Transaction on Image Processing ( February 1997), based on alternating 
minimizations. Other methods may also be used. 



This new criterion therefore suppresses the background block effects, and 
simultaneously segments moving objects. 

This criterion provides a sequence of moving card elements. In order to be able to 
treat each element separately, they must be spatially isolated from one another. However, 
the more numerous the block effects are in the original sequence, the more the c* cards 
present false or poor quality information. For example, a DCT block whose intensity 
changes from one image to the next may be thought of as a moving object. Several pre- 
treatments are therefore necessary before isolating each element: 

Thresholding of the c* card. The values with intensity less than a given threshold 

are brought to 0, and the others to 1 . 

Mathematical closure and filling of each object. Mathematical closure occurs, in 
other words dilatation followed by erosion, by a structuring element of size nxn, 
preferably with n=3. The element is filled using a traditional image path method. 
Other methods can also be used, such as active geodesic contours. 
Mathematical opening and suppression of certain objects. The opening consists of 
making an erosion followed by a dilatation, to suppress the false elements coming 
from the DCT blocks. Other methods can also be used. 

From these c* it is possible to label each element, isolate them from one another 
and consider them as objects. Henceforth, each treatment described will be completed 
independently on each object. 

For each object in the sequence, certain characteristics will be determined which 
will permit a detailed and adapted treatment: 

Evolution of the shape, average height and size, position, barycenter, in the 

sequence. 

Object spatial segmentation information. A given object is spatially segmented to 
determine the different zones it contains (discontinuities, homogeneous zones...). 
Traditional methods for spatial segmentation of fixed images can be used. 



Estimation of object movement using traditional "block-matching" methods, or 

optical flow. This estimation of movement provides a movement vector 

d=(dx if dyi) for each object, and for each image / of the sequence. 

Once each object has been isolated, and its movement determined, its treatment 
can be customized to suppress the block effects it contains. This phase may be performed 
in parallel on each object, to optimize speed of execution. 

For each object, we look for: 

O k * = arg mm(J u ) 

with 

JxiPu) = Jx (O k ) +X 2 2 J 2 (O k ) + ti 2 V 3 (0*) + Mi V 4 (0*) (7) 
where 

• J\(O k )= Yj J (A" ! ) 2 (O k -Pkn(x + ch k+h y + (fy kH )) 2 is a tempore^ 

'=» n 

averaging of the object, with compensation for movement. The value of n depends upon 
the object characteristics, in particular its non-stationary nature. The more rapidly the 
object evolves over time, the smaller the n chosen will be. 

• Ji(Ok) = J (ck - l) 2 <t>3 ( || VOa: [] )) regularizes the object. ^ is adaptive; it 

a 

depends upon the spatial segmentation chosen to determine the different object zones, 
and permits customizing the object treatment. 



J 3 (O k )= J (c*-l) 2 T 

MO k ) = v*l (c k -iy 



8 ; 



suppresses the block effects on the object. 



DOk-pk + l-DOk + pk-^ 2 



+ Va\ ( Ck -\f[^DOk-pk-l-DOk + pk-^ 2 

permits restricting each pixel of each object to the quantification interval, to reduce 
quantification noise on the object. 
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An optimal solution Ok* is obtained for — — = 0, equivalent to: 



(ck-iy 



^ (Ok - pk + i) - A.2 1 div\ 



. ^'(Ival) 



|Vft| 



ROk + ^D'KiOk) 



= 0 



with 



k(/) 



f 



DO k -p k +l 
DO k -p k -l 



0 



if Z)0*</?*-J 



if D<9*>/>*+| 

if DO k e\j> k -l;p k +l] 



The method of resolution used to solve the equation (8) is identical to that used 

above. 

The method presented may be simplified, in order to reduce its complexity, and 
therefore calculation time. 

Simplification during estimation of background and c*. 

A first simplification consists of setting r|i=0 in the equation (3). The wavelet 
coefficient thresholding can in this case be used as a pretreatment for each image entering 
the sequence. \i } =0 is also posited in (3), and the interval restriction can be implemented 
by projection on the quantification intervals. The result of this simplification is a 
significant decrease in calculation time, at the cost of a slight decrease in quality. 

The second simplification consists of suppressing regularization on the c* , or 

positing Yi=0 in (3). To obtain the c k sequence, -r^|r. = o = 0 is solved (cf. equation (6)). 

OCk 



An explicit formula a* = 77 is then obtained which permits calculation of 

<* + \f-P*) 

the sequence of moving objects. 

Equation (7) can be simplified: 

by positing X2 in (7), object regularization is suppressed. In this case, only 
temporal averaging of the object occurs, with compensation for movement, 
by positing r| 2 = 0 in the equation (7) and performing thresholding as a 
pretreatment of each object. 

by positing |i 2 = 0 in (7), and performing projection on the qualification intervals 
of each object. 

By totaling some of these simplifications, the algorithm becomes quick, and may 
be adapted to real time applications. 

The decoded sequence p is reconstituted by using a background image over a 
duration of N images, and projecting into it the M objects: 

p* = c k * 2 J*+(c k *-l) 2 O k * (9) 

If, for a given pixel, you are on an object, c**=0, the Ok pixel is then projected, 
otherwise c**=l and the pixel is projected from background /*. 

The details of one or more embodiments of the invention are set forth in the 
accompanying description above. Although any methods and materials similar or 
equivalent to those described herein can be used in the practice or testing of the present 
invention, the preferred methods and materials are now described. Other features, objects, 
and advantages of the invention will be apparent from the description and from the 
claims. In the specification and the appended claims, the singular forms include plural 
referents unless the context clearly dictates otherwise. Unless defined otherwise, all 
technical and scientific terms used herein have the same meaning as commonly 
understood by one of ordinary skill in the art to which this invention belongs. All patents 
and publications cited in this specification are incorporated by reference. 
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The foregoing description has been presented only for the purposes of illustration 
and is not intended to limit the invention to the precise form disclosed, but by the claims 
appended hereto. 
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